System and method for generating emoji mashups with machine learning

ABSTRACT

Aspects of the present disclosure involve systems, methods, devices, and the like for emoji mashup generation. The system and method introduce a method and model that can generate emoji mashups representative of contextual information received by a user at an application. The emoji mashup may come in the form of two or more emojis coherently combined to represent the contextual idea or emotion being conveyed.

TECHNICAL FIELD

The present disclosure generally relates to user device communication,and more specifically, to user device communication using emojis.

BACKGROUND

Nowadays with the evolution and proliferation of devices, users areconstantly connected to the internet and social media as a means forcommunication. Oftentimes, in the communication the users resort to theuse of emojis to express an emotion, an idea, place, event, etc. Theemojis are often available for selection from the application in use andmay be selected by the user. In some instances however, the emoji mayappear in response to the word or group of words typed by the user.These emojis are often restricted to the emojis available to theapplication and/or constraint by the one or more words identified by theapplication that relate to an emoji. This however, may lead to anincorrect emoji being presented, as the emoji may not fit the occasion.In other words, the emojis presented are constrained to the one or morewords matched to the emoji. Thus, the sentiment or occasion as describedby a sentence typed is not understood and the user instead resorts to asticker or gif for the emotion. Therefore, it would be beneficial tocreate a system that can generate emojis that are tailored for theconversation.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a graphical diagram of a user communicating with auser device.

FIG. 2 illustrates a graphical diagram of user device communicationusing emoji mashup.

FIG. 3 illustrates a block diagram of a system for the overall processfor generating emoji mashups with machine learning.

FIG. 4 illustrates a block diagram exemplifying a training andpredicting method for generating emoji mashups with machine learning.

FIGS. 5A-5B illustrate graphical diagrams demonstrating exemplary emojimashups.

FIGS. 6A-6B illustrate graphical diagrams of training and retrievalprocesses for generating emoji mashups.

FIGS. 7A-7B illustrate other graphical diagrams of training andretrieval processes for generating emoji mashups.

FIG. 8 illustrates a flow diagram illustrating operations for generatingemoji mashups.

FIG. 9 illustrates a block diagram of a system for generating emojimashups with machine learning.

FIG. 10 illustrates an example block diagram of a computer systemsuitable for implementing one or more devices of the communicationsystems of FIGS. 1-9.

Embodiments of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures, whereasshowings therein are for purposes of illustrating embodiments of thepresent disclosure and not for purposes of limiting the same.

DETAILED DESCRIPTION

In the following description, specific details are set forth describingsome embodiments consistent with the present disclosure. It will beapparent, however, to one skilled in the art that some embodiments maybe practiced without some or all of these specific details. The specificembodiments disclosed herein are meant to be illustrative but notlimiting. One skilled in the art may realize other elements that,although not specifically described here, are within the scope and thespirit of this disclosure. In addition, to avoid unnecessary repetition,one or more features shown and described in association with oneembodiment may be incorporated into other embodiments unlessspecifically described otherwise or if the one or more features wouldmake an embodiment non-functional.

Aspects of the present disclosure involve systems, methods, devices, andthe like for emoji mashup generation. In one embodiment, a system isintroduced that generates emoji mashups representative of contextualinformation received by a user at an application. The emoji mashup maycome in the form of two or more emojis coherently combined to representthe contextual idea or emotion being conveyed.

Conventionally, device users have depended on predefined emojis for usein expressing emotions, ideas, or places. However, oftentimes, the usermay be limited to those emojis available on the application. In someinstances, the emoji available is further limited and presented inresponse to the recognition of one or more words being input/typed bythe user. The emojis presented however, may be out of context, as thepresented emojis are limited to those on the application and/or thecorrelation between the one or more words detected.

An example, FIG. 1 illustrates a graphical diagram 100 of a user 104communicating with a user device 102. In particular, FIG. 1 illustratesa user 104 interacting with an application 106 on a user device 102. Theuser device 102 may include a smart phone, tablet, computer, laptop,wearable device, tablet, virtual reality system, etc. The user device102 may be capable of communicating with one or more devices over one ormore networks. In some instances, the user device 102 may include theinteraction with an application for communicating over social media,communicating with another individual, transferring funds, processingtransactions or the like. As illustrated, the user 104 may beinteracting with the application (e.g., Venmo) over user device 102. Theapplication (e.g., Venmo) 106 may include one that presents emojis inresponse to detecting a word(s) (e.g., dinner) that is correlated withthe words detected. As an example, FIG. 1 illustrates user 104interacting with Venmo with the indication that a payment will be madeto a person (e.g., Robert Kuo) in response to a dinner outing. In themessage, the user is thanking Robert for dinner and in response to thedetection of the word dinner 108, emojis 110 that have been identifiedto correlate with the word dinner 108 appear. In the example, theapplication recognizes the word dinner 108 and in response silverwareand dinner plate emojis 110 are presented. A disadvantage however, asillustrated, is that word-to-emoji type matching is limited by thecorrelations that the system has and corresponding emojis 110 availablefor selection.

In one embodiment, a system and method is introduced that enables emojimashups with machine learning. That is to say, a system and method areintroduced that enable the ability to combine two or more emojis togenerate at least a single emoji that represents more than a single wordor words, but instead a sentence and/or the context involved in acommunication.

FIG. 2 illustrates a graphical diagram of user device communicationusing emoji mashup. In particular FIG. 2 illustrates a flow diagram 200for generating emoji mashups that may then be presented on a userinterface of an application 106 on a user device 102.

As indicated above, a large limitation exists in current emojis used,based in part, on the partial library that may be available on theapplication as well as the strict emoji-word designation based onpredefined correlations. Flow diagram 200 is presented as an exemplarycommunication that can occur between various systems that can enable thegeneration of emoji mashups that are more closely related to thecontextual information on the message at the time.

For example, as illustrated in flow diagram 200, an emojirepository/system 202 that may be in communication with externalnetworks 210 including social networks 208 (e.g., Twitter, Facebook) maybe used. These networks can communicate and share emojis available thatmay be used by the user device 102 during an interaction with anotheruser (e.g., a Venmo transaction). The flow diagram also illustrates acoordinate system 204 that may be in communication with at least theuser device 104 and external networks 210. The coordinate system 204 canbe a system that uses the emoji details gathered from the user device102, social networks 208, and other external networks 210 to determinehow to best locate the two or more emojis that become the emoji mashup212 and/or to extract coordinate information from two or more emojis todetermine how to best place the emojis with respect to each other inresponse to the contextual information gathered during the user deviceinteraction with the another user.

The coordination, repository and use of intelligent algorithms workjointly to generate the new emoji mashup. Feedback from user input isalso used to learn and identify the best matches as well as identify newemoji mashups 212. To illustrate the idea of emoji mashup 212, FIG. 2includes a conversation on a display 216 on user device 102. As anexample, on the display 216, user 104 is communicating with another userregarding a concert that was attended. The other user comments on theconcert and how well it went while user 104 responds with, “IKR?! Dude,Dan was a friggin rock star . . . ” In response the user's comments 214,emoji mashups 214 are presented which are suggested in response to thecontext in the conversation.

Notice that unlike conventional systems where the word “rock” or “star”would be recognized and a rock or star would be suggested, here insteadmore relevant emoji mashups 214 are suggested. For example, a guitar anda star are combined, a rock&roll emoji and stars are combined, and atrophy and music notes are combined. Thus, the emoji mashup 214 presentsan emoji that represents the idea and/or emotion in a more comprehensivemanner.

Turning to FIG. 3, a more detailed description of the overall processfor generating the emoji mashup 214 is presented. In particular, FIG. 3illustrates a block diagram of a methodology 300 for generating emojimashups. Methodology 300 presents the model that may be used forretrieving, analyzing, and training information received for suggestingan emoji mashups 212. Methodology 300 can begin with first analyzing thetext 302 received by the application. As indicated above, an aspect ofthe current embodiment includes retrieving mashup emojis 212 thatclosely correlate with the contextual information received. Analyzingthe text can include taking the message transmitted and evaluating thecontextual information to identify the words transmitted. In addition,the information can also be run through a system and method ofevaluating the context which includes word vectorization 304. Wordvectorization 304 is a method that uses computational intelligenceincluding neural networks and natural language processing for generatingdictionaries from words into real-valued vectors. The process whichincludes converting a bitmap image into a vector representation suchthat terms are given a corresponding number such that the closer thewords are in relationship to each other the closer the numerical numberbetween the words. As an example, the words can be vectorized to findsimilar meanings using language models to extract word functions andstructure. FIG. 4 and its corresponding description below provides anexample of word vectorization. In addition to vectorization, theinformation may be sent through a sentiment analyzer where clues aboutthe message tone, purpose, and context may be used in the analysis.

After the words are vectorized, the words can be filtered 306. Thefiltered vectors may be converted into matrices which can be used toprogrammatically generate new emojis 308. The new emojis generated canbe emojis identified from social networks, system repositories, othernetworks, etc. Once the new emojis are generated and/or retrieved, theemojis are combined. Combining the emojis can occur by using matchmakinglogic 310. The matchmaking logic can include coordinates, imagerecognition systems, as well as machine learning algorithm which can beused to learn and/or determine how to combine the emojis coherently 312.For example, coordinates from each set of emojis retrieved or generated308 can be analyzed to determine their corresponding center to determinehow to best combine. Once one or more emojis are combined to generateemoji mashups 212, the emojis can be presented to the user 104 forselection.

To illustrate methodology 300, consider a user 104 whose input includes“Dude, you're on fire!” For such input, methodology can use wordvectorization and sentiment analysis to determine that the communicationincludes a positive sentiment and smiley face and flame emoji can beretrieved. Once these two emojis are retrieved, the matchmakingalgorithm can be used to determine how to scale and place the emojisrelative to each other. In one embodiment, the face emoji may be placedprominently in front of the flame emoji which can sit behind on by thehead portion of the face. Additionally, a refresh button may beavailable which can be used to generate a new emoji mashup. Thus, withthe use of the external networks, user feedback, and system userbase(e.g., Paypal/Venmo userbase), machine learning and neural networks maybe used to generate new and improved emoji mashups 212 over time.

To illustrate some of the processes involved in methodology 300 forgenerating the emoji mashups 212, FIGS. 4-5 are included. Turning toFIG. 4, word training and prediction is illustrated. In particular, FIG.4 illustrates a block diagram 400 exemplifying a training and predictingmethod for generating emoji mashups with machine learning.

As previously indicated, word vectorization 304 is a technique that maybe used in the text analysis for identifying and predicting emojis basedon the written text. As an example of how word vectorization may occur,word2vec training model is presented in FIG. 4. Word2vec is a model thatmay be used to illustrate a model that may be used to learn to mapdiscrete words into a low-dimensional vector. Generally, word2vec may bean algorithm that may comprise at least two models that are trained topredict relationships. In particular, vector space models such asword2vec enable words to be represented in a continuous vector spacesuch that similar words are maps to nearby data points through the useof machine learning. As an example, consider the word “animal”, the datapoint assigned to it will be very similar to that which would beassigned to a word like “cat” or “dog.” As another example, the word“man” would have a closer value to “king” than the word “queen.” In someinstances, an open source library may be used to obtain these values.

Returning to FIG. 4, an illustration of the implementation of theword2vec model is illustrated. Note that the word2vce model in thecurrent embodiment is used to train the entire sentence. Unlikeconventional systems which simply map to a single word, the currentsystem learns the entire sentence 402 in order to predict the two ormore emojis that best represent the idea or emotion being communicated.As an example, training diagram 420 illustrates the sentence “you are arock star” such that the system presents the guitar and star as asuggestion. This training process continues for commonly used phrasesand statements and builds as the communication, suggestion and feedbackis received from the user. Again, the training and emojis are suggestedbased using machine learning algorithms, libraries, databases, andmodels including but not limited to word2vec, open source libraries anddatabases, convolutional neural networks, supervised learning,clustering, dimensionality reduction, structured prediction, decisiontree learning and the like. As the phrases/sentences are learned, thesystem may begin making predictions as illustrated in predicting diagram440. Notice that now the system recognizes the trained sentence 408,“You are a rock star!” and produces various emoji suggestions withcorresponding data points 410 with the suggestion of a guitar and a starreceiving the highest data points of 0.86 and 0.77 respectively. Oncethe emojis are identified or new emojis generated 308, as indicatedabove, the method continues to combine the emojis using matchmakinglogic.

A large part of the matchmaking logic includes determining how to mergethe emojis. In one embodiment, to determine how to merge the emojis,object recognition is used to determine what the image is and an optimallocation to merge. For example, if the two emojis identified include asmiley face and fire in response to “Dude you are on fire!”understanding how to merge the smiley face and the fire is determinedusing object recognition. To accomplish this, in one embodiment, thesystem can perform a center of mass like analysis to determine where thecenter of the object is. In another embodiment, the system can recognizethe object (e.g., smiley face and fire images) and extract theircoordinates. In one example, the dimensions may be pulled from theobject while in other instances, the images may be turn a determinedamount such that depth is provided to the image and coordinates can beextracted. The coordinates can then be provided to the matchmaking logicwhich can in-turn and suggest various ways to merge the two imagesdetected (e.g., smiley face and fire emojis).

FIGS. 5A-5B illustrate graphical diagrams demonstrating exemplary emojimashups. In particular, FIG. 5A illustrates three sample smiley faceemojis that are merged with the fire emoji. Referring to middle smileyfire emoji 502, note that coordinates for the smiley face and fire emojiare presented and referenced for the placement of the two to generatethe emoji mashup 212. FIG. 5B illustrates a guitar emoji 504 and a staremoji 506 and the generation of the rockstar guitar emoji 508. Noticethat the guitar emoji 504 and star emoji 506 were presented in responseto high data points that may have been determined from the initialtraining by the machine learning system. Also notice that in thisinstance, the emojis resulted in a newly generated emoji (e.g., rockstarguitar emoji 508). Note that in some instances, the final emoji maybe aproduct of the combination of the two or more emojis suggested by thelearning system while in other instances, the emojis may be a newlygenerated emoji defined by the two or more emojis initially identified.The newly generated emoji may be a merger of the two or a previouslydefined emoji located internally or on an external network. Also notethat the emojis may have been obtained from an external library,repository, database or the like.

FIGS. 6A-7B illustrate graphical diagrams of training and retrievalprocesses for generating emoji mashups. FIGS. 6A-6B illustrate earlyprocesses where the system may include minimal or no user feedback datathat can help train and tailor the emojis that are suggested. FIGS.7A-7B illustrate a more mature process where the system is not onlyusing external information (e.g., Twitter® feeds, Google libraries,etc.), but also using internal user feedback to train and present theuser 104 with emoji mashup suggestions.

Turning to FIG. 6A, system training is presented, in particular texttraining process 600 begins with training text sequences (e.g., a textmessage on a messaging application) for emoji representation. To beginthe emoji training process, external sources are used for the retrievalof information. For example, social media feeds 602 (e.g., Twitterfeeds) may be collected for the use in training the system. The socialmedia feeds may then go through some data processing 604, where thefeeds and other information collected is in preparation for the word2vecconversion 606. As indicated above, word2vec 606 is a training modelthat may be used to convert words into vectors where part of word2vec606 entails turning the text into vectors and extracting the text andthe emojis. Thus, input 608 word vectors produces the emojis which areused as the labels. This process is performed for the sequences suchthat a text2emoji sequence is identified in the translation model 610.

FIG. 6A then continues the training process with emoji training process620 where the emoji images are trained using object detection. In emojitraining process 620, the emoji images identified are then processed 614and resized using the object detection model 618. Note that emoji imagesare used for exemplary purposes and are not so restricted, as otherforms and images may be contemplated.

At FIG. 6B, the runtime process 650 is presented. As illustrated inruntime process 650, begins with the receipt of user input 622. Userinput can come in the form of a phrase, a sentence, an abbreviation,text, or the like. Oftentimes, the user input will be received at thedisplay of a user device 102 while on an application 216 incommunication with one or more other users.

In runtime process 650, like text training process 600, the text isprocesses and converted using word2vec 626. Next, the information is runagainst the pre-trained Text2Emoji model 630 which can then output theemoji sequences 632 that are correlated to the input text from the user104. In some embodiments, the emoji sequences identified can then bepresented to the user 104 on the user device UI 634 for user observationand selection 646. Additionally or alternatively, the emoji sequences632 obtained can are received 636, processed 638, and run against thepre-trained object detection model 640. After the pre-trained objectdetection model 640, emoji coordinates may be evaluated 642 andextracted and sent to the user UI for user selection. Note that in someinstances, the two or more emoji sequences may be presented to the user104, while in other instances, multiple mashup emojis may already begenerated based on the coordinates 642 extracted and presented to theuser for user selection 646. After the user 104 has made a selection asto a preferred emoji arrangement, emoji pair, emoji overlay and/or emojimashup user feedback is stored 648.

As more user preferences and selections are acquired by the system, thenuser feedback may be used to generate the emoji mashups. For example, intraining and runtime processes of FIGS. 7A-7B, a system which includesuser feedback is presented. Notice that text training process 700 andemoji training process 720 are very similar to training process 600 and620 corresponding to FIGS. 6A-6B. As such, the processes descriptionsare similar with the exception of user feedback 712, 714, which are nowused to train both the text model and the positioning model. Similarly,runtime process 750 of FIG. 7B imitates runtime process 650 of FIG. 6B.Distinct from runtime process 650 however, is the use of positioningmodel 736 of FIG. 7B. The positioning model was training during emojitraining process 720 with the use of user feedback, therefore, atruntime process 750, the pre-trained object detection model 640 is nolonger used and instead replaced by the positioning model 736. Oncecoordinate information 738 is extracted from the emoji sequences againthe emojis and coordinates can be presented to the user for userselection 742 and feedback is stored 744.

Note that the process presented is for exemplary purposes and otherprocesses, modeling, and training may be contemplated. In addition, apreviously indicated, the system is not restricted to the use of theword2vec model as other machine learning models may be used.Additionally, the system is not restricted to the use of emojis andother forms and types of images may be contemplated.

FIG. 8 illustrates an example process for generating emoji mashups thatmay be implemented in on a system, such as system 900 and device 102 inFIGS. 9 and 1 respectively. In particular, FIG. 8 illustrates a flowdiagram illustrating operations for receiving a user input and inresponse presenting the user with emoji mashup suggestions to representthe emotion or idea conveyed in the user input. According to someembodiments, process 800 may include one or more of operations 802-818,which may be implemented, at least in part, in the form of executablecode stored on a non-transitory, tangible, machine readable media that,when run on one or more hardware processors, may cause a system toperform one or more of the operations 802-818.

Process 800 may begin with operation 802, where user input is received.The user input may be in the form of a sequence, statement, sentence,phrase or other text. The user input may be input into an applicationused for transacting, communicating, or interacting with another user.For example, the user input may be at an application like Venmo where auser may send or request a payment to another user with a briefstatement and/or emoji regarding the transaction involved.

The user input received is then analyzed for identifying a relevantemoji(s) to present. To analyze the text, at operation 804, the inputtext is vectorized. Word vectorization may occur using a model such asbut not limited to word2vec, where word2vec may be an algorithm that maycomprise at least two models that are trained to predict relationships.Thus, in this instance, the user sequence is vectorized correspondingemoji(s) are allocated based on the word. As the entire sequence (userinput) is vectorized, the model is trained to extract a series of emojisequences, in operation 806, that correspond to the context of thesequence input. In some instances, social media networks may be used toidentify the sequences and/or emojis for use, while in other instancesuser selection feedback may exist such that the emojis extracted areretrieved from an existing database or other storage unit. Note that inaddition to vectorization, the information may be sent through asentiment analyzer where clues about the message tone, purpose, andcontext may be used in the analysis and in identifying the emojisequences.

If the system model used for extracting the emoji model is mature andsufficient user feedback exists, then a positioning model exists withuser preferences and emoji details. As such, at operation 808, adetermination is made as to whether a positioning model is available. Ifthe system is still underdeveloped or if a new sequence is identified,the emoji processing may continue further using an object detectionmodel at operation 810. Object detection model may be a model used todetect the emoji sequences extracted from the text input such thatcoordinate information may be extract from the emojis at operation 814.Alternatively, at operation 812 if the input received is recognizedand/or sufficient user feedback exists such that emoji training is notneeded, then emoji sequences may be processed through a emoji positingmodel at operation 812 so that coordinate information may be extractedat operation 814. Once the coordinate information is known at operation814, then the two or more emojis identified (emoji sequences) may becoherently combined to generate an emoji mashup representative of theinput received. The emoji mashup(s) coherently combined may occur atoperation 816 where the output mashup emoji(s) may be presented to theuser for selection. Note, however, that in some instances, the emojisequence at operation 806 may additionally or alternatively be presentedto the user for the opportunity to combine the emojis by the user.

Note that more or fewer operations may exist in performing method 800.In addition, an operation may exist for determining new emoji or othermedia object mashup. In addition, the operations are not limited to thetraining models identified. Further, user selection may be stored forlater use by the user and/or another user.

FIG. 9 illustrates, in block diagram format, an example embodiment of acomputing environment adapted for implementing a system for generatingemoji mashups. As shown, a computing environment 900 may comprise orimplement a plurality of servers and/or software components that operateto perform various methodologies in accordance with the describedembodiments. Severs may include, for example, stand-alone andenterprise-class servers operating a server operating system (OS) suchas a MICROSOFT® OS, a UNIX® OS, a LINUX® OS, or other suitableserver-based OS. It may be appreciated that the servers illustrated inFIG. 4 may be deployed in other ways and that the operations performedand/or the services provided by such servers may be combined,distributed, and/or separated for a given implementation and may beperformed by a greater number or fewer number of servers. One or moreservers may be operated and/or maintained by the same or differententities.

Computing environment 900 may include, among various devices, servers,databases and other elements, one or more clients 902 that may compriseor employ one or more client devices 904, such as a laptop, a mobilecomputing device, a tablet, a PC, a wearable device, and/or any othercomputing device having computing and/or communications capabilities inaccordance with the described embodiments. Client devices 904 mayinclude a cellular telephone, smart phone, electronic wearable device(e.g., smart watch, virtual reality headset), or other similar mobiledevices that a user may carry on or about his or her person and accessreadily.

Client devices 904 generally may provide one or more client programs906, such as system programs and application programs to perform variouscomputing and/or communications operations. Some example system programsmay include, without limitation, an operating system (e.g., MICROSOFT®OS, UNIX® OS, LINUX® OS, Symbian OS™, Embedix OS, Binary Run-timeEnvironment for Wireless (BREW) OS, JavaOS, a Wireless ApplicationProtocol (WAP) OS, and others), device drivers, programming tools,utility programs, software libraries, application programming interfaces(APIs), and so forth. Some example application programs may include,without limitation, a web browser application, messaging applications(e.g., e-mail, IM, SMS, MMS, telephone, voicemail, VoIP, videomessaging, interne relay chat (IRC)), contacts application, calendarapplication, electronic document application, database application,media application (e.g., music, video, television), location-basedservices (LBS) applications (e.g., GPS, mapping, directions, positioningsystems, geolocation, point-of-interest, locator) that may utilizehardware components such as an antenna, and so forth. One or more ofclient programs 906 may display various graphical user interfaces (GUIs)to present information to and/or receive information from one or moreusers of client devices 904. In some embodiments, client programs 906may include one or more applications configured to conduct some or allof the functionalities and/or processes discussed above and inconjunction FIGS. 1-8.

As shown, client devices 904 may be communicatively coupled via one ormore networks 908 to a network-based system 910. Network-based system910 may be structured, arranged, and/or configured to allow client 902to establish one or more communications sessions between network-basedsystem 910 and various computing devices 904 and/or client programs 906.Accordingly, a communications session between client devices 904 andnetwork-based system 910 may involve the unidirectional and/orbidirectional exchange of information and may occur over one or moretypes of networks 908 depending on the mode of communication. While theembodiment of FIG. 9 illustrates a computing environment 900 deployed ina client-server operating relationship, it is to be understood thatother suitable operating environments, relationships, and/orarchitectures may be used in accordance with the described embodiments.

Data communications between client devices 904 and the network-basedsystem 910 may be sent and received over one or more networks 508 suchas the Internet, a WAN, a WWAN, a WLAN, a mobile telephone network, alandline telephone network, personal area network, as well as othersuitable networks. For example, client devices 904 may communicate withnetwork-based system 910 over the Internet or other suitable WAN bysending and or receiving information via interaction with a web site,e-mail, IM session, and/or video messaging session. Any of a widevariety of suitable communication types between client devices 904 andsystem 910 may take place, as will be readily appreciated. Inparticular, wireless communications of any suitable form may take placebetween client device 904 and system 910, such as that which oftenoccurs in the case of mobile phones or other personal and/or mobiledevices.

In various embodiments, computing environment 900 may include, amongother elements, a third party 912, which may comprise or employthird-party devices 914 hosting third-party applications 516. In variousimplementations, third-party devices 514 and/or third-party applications916 may host applications associated with or employed by a third party912. For example, third-party devices 914 and/or third-partyapplications 916 may enable network-based system 910 to provide client902 and/or system 910 with additional services and/or information, suchas merchant information, data communications, payment services, securityfunctions, customer support, and/or other services, some of which willbe discussed in greater detail below. Third-party devices 914 and/orthird-party applications 916 may also provide system 910 and/or client902 with other information and/or services, such as email servicesand/or information, property transfer and/or handling, purchase servicesand/or information, and/or other online services and/or information.

In one embodiment, third-party devices 914 may include one or moreservers, such as a transaction server that manages and archivestransactions. In some embodiments, the third-party devices may include apurchase database that can provide information regarding purchases ofdifferent items and/or products. In yet another embodiment, third-partysevers 914 may include one or more servers for aggregating consumerdata, purchase data, and other statistics.

Network-based system 910 may comprise one or more communications servers920 to provide suitable interfaces that enable communication usingvarious modes of communication and/or via one or more networks 908.Communications servers 920 may include a web server 922, an API server924, and/or a messaging server 926 to provide interfaces to one or moreapplication servers 930. Application servers 930 of network-based system910 may be structured, arranged, and/or configured to provide variousonline services, merchant identification services, merchant informationservices, purchasing services, monetary transfers, checkout processing,data gathering, data analysis, and other services to users that accessnetwork-based system 910. In various embodiments, client devices 904and/or third-party devices 914 may communicate with application servers930 of network-based system 910 via one or more of a web interfaceprovided by web server 922, a programmatic interface provided by APIserver 924, and/or a messaging interface provided by messaging server926. It may be appreciated that web server 922, API server 924, andmessaging server 526 may be structured, arranged, and/or configured tocommunicate with various types of client devices 904, third-partydevices 914, third-party applications 916, and/or client programs 906and may interoperate with each other in some implementations.

Web server 922 may be arranged to communicate with web clients and/orapplications such as a web browser, web browser toolbar, desktop widget,mobile widget, web-based application, web-based interpreter, virtualmachine, mobile applications, and so forth. API server 924 may bearranged to communicate with various client programs 906 and/or athird-party application 916 comprising an implementation of API fornetwork-based system 910. Messaging server 926 may be arranged tocommunicate with various messaging clients and/or applications such ase-mail, IM, SMS, MMS, telephone, VoIP, video messaging, IRC, and soforth, and messaging server 926 may provide a messaging interface toenable access by client 902 and/or third party 912 to the variousservices and functions provided by application servers 930.

Application servers 930 of network-based system 910 may be a server thatprovides various services to clients including, but not limited to, dataanalysis, geofence management, order processing, checkout processing,and/or the like. Application server 930 of network-based system 910 mayprovide services to a third party merchants such as real time consumermetric visualizations, real time purchase information, and/or the like.Application servers 930 may include an account server 932, deviceidentification server 934, payment server 936, content selection server938, profile merging server 940, user ID server 942, feedback server954, and/or content statistics server 946. Note that any one or more ofthe serves 932-946 may be used in storing and/or retrieving emojis, userfeedback, coordinates, emoji positioning, etc. For example, userselections may be stored in feedback server 944. These servers, whichmay be in addition to other servers, may be structured and arranged toconfigure the system for monitoring queues and identifying ways forreducing queue times.

Application servers 930, in turn, may be coupled to and capable ofaccessing one or more databases 950 including a profile database 952, anaccount database 954, geofence database 956, and/or the like. Databases950 generally may store and maintain various types of information foruse by application servers 930 and may comprise or be implemented byvarious types of computer storage devices (e.g., servers, memory) and/ordatabase structures (e.g., relational, object-oriented, hierarchical,dimensional, network) in accordance with the described embodiments.

FIG. 10 illustrates an example computer system 1000 in block diagramformat suitable for implementing on one or more devices of the system inFIGS. 1-9. In various implementations, a device that includes computersystem 1000 may comprise a personal computing device (e.g., a smart ormobile device, a computing tablet, a personal computer, laptop, wearabledevice, PDA, user device 102, etc.) that is capable of communicatingwith a network 926 (e.g., networks 208, 210). A service provider and/ora content provider may utilize a network computing device (e.g., anetwork server) capable of communicating with the network. It should beappreciated that each of the devices utilized by users, serviceproviders, and content providers may be implemented as computer system1000 in a manner as follows.

Additionally, as more and more devices become communication capable,such as new smart devices using wireless communication to report, track,message, relay information and so forth, these devices may be part ofcomputer system 1000. For example, windows, walls, and other objects maydouble as touch screen devices for users to interact with. Such devicesmay be incorporated with the systems discussed herein.

Computer system 1000 may include a bus 1010 or other communicationmechanisms for communicating information data, signals, and informationbetween various components of computer system 1000. Components includean input/output (I/O) component 1004 that processes a user action, suchas selecting keys from a keypad/keyboard, selecting one or more buttons,links, actuatable elements, etc., and sending a corresponding signal tobus 1010. I/O component 1004 may also include an output component, suchas a display 1002 and a cursor control 1008 (such as a keyboard, keypad,mouse, touchscreen, etc.). In some examples, I/O component 1004 otherdevices, such as another user device, a merchant server, an emailserver, application service provider, web server, a payment providerserver, and/or other servers via a network. In various embodiments, suchas for many cellular telephone and other mobile device embodiments, thistransmission may be wireless, although other transmission mediums andmethods may also be suitable. A processor 1018, which may be amicro-controller, digital signal processor (DSP), or other processingcomponent, that processes these various signals, such as for display oncomputer system 1000 or transmission to other devices over a network1026 via a communication link 1024. Again, communication link 1024 maybe a wireless communication in some embodiments. Processor 1018 may alsocontrol transmission of information, such as cookies, IP addresses,images, and/or the like to other devices.

Components of computer system 1000 also include a system memorycomponent 1012 (e.g., RAM), a static storage component 1014 (e.g., ROM),and/or a disk drive 1016. Computer system 1000 performs specificoperations by processor 1018 and other components by executing one ormore sequences of instructions contained in system memory component 1012(e.g., text processing and emoji processing). Logic may be encoded in acomputer readable medium, which may refer to any medium thatparticipates in providing instructions to processor 1018 for execution.Such a medium may take many forms, including but not limited to,non-volatile media, volatile media, and/or transmission media. Invarious implementations, non-volatile media includes optical or magneticdisks, volatile media includes dynamic memory such as system memorycomponent 1012, and transmission media includes coaxial cables, copperwire, and fiber optics, including wires that comprise bus 1010. In oneembodiment, the logic is encoded in a non-transitory machine-readablemedium. In one example, transmission media may take the form of acousticor light waves, such as those generated during radio wave, optical, andinfrared data communications.

Some common forms of computer readable media include, for example, harddisk, magnetic tape, any other magnetic medium, CD-ROM, any otheroptical medium, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip orcartridge, or any other medium from which a computer is adapted to read.

Components of computer system 1000 may also include a short rangecommunications interface 1020. Short range communications interface1020, in various embodiments, may include transceiver circuitry, anantenna, and/or waveguide. Short range communications interface 1020 mayuse one or more short-range wireless communication technologies,protocols, and/or standards (e.g., WiFi, Bluetooth®, Bluetooth LowEnergy (BLE), infrared, NFC, etc.).

Short range communications interface 1020, in various embodiments, maybe configured to detect other devices (e.g., device 102, secondary userdevice, etc.) with short range communications technology near computersystem 1000. Short range communications interface 1020 may create acommunication area for detecting other devices with short rangecommunication capabilities. When other devices with short rangecommunications capabilities are placed in the communication area ofshort range communications interface 1020, short range communicationsinterface 1020 may detect the other devices and exchange data with theother devices. Short range communications interface 1020 may receiveidentifier data packets from the other devices when in sufficientlyclose proximity. The identifier data packets may include one or moreidentifiers, which may be operating system registry entries, cookiesassociated with an application, identifiers associated with hardware ofthe other device, and/or various other appropriate identifiers.

In some embodiments, short range communications interface 1020 mayidentify a local area network using a short range communicationsprotocol, such as WiFi, and join the local area network. In someexamples, computer system 1000 may discover and/or communicate withother devices that are a part of the local area network using shortrange communications interface 1020. In some embodiments, short rangecommunications interface 1020 may further exchange data and informationwith the other devices that are communicatively coupled with short rangecommunications interface 1020.

In various embodiments of the present disclosure, execution ofinstruction sequences to practice the present disclosure may beperformed by computer system 1000. In various other embodiments of thepresent disclosure, a plurality of computer systems 1000 coupled bycommunication link 1024 to the network (e.g., such as a LAN, WLAN, PTSN,and/or various other wired or wireless networks, includingtelecommunications, mobile, and cellular phone networks) may performinstruction sequences to practice the present disclosure in coordinationwith one another. Modules described herein may be embodied in one ormore computer readable media or be in communication with one or moreprocessors to execute or process the techniques and algorithms describedherein.

A computer system may transmit and receive messages, data, informationand instructions, including one or more programs (i.e., applicationcode) through a communication link 1024 and a communication interface.Received program code may be executed by a processor as received and/orstored in a disk drive component or some other non-volatile storagecomponent for execution.

Where applicable, various embodiments provided by the present disclosuremay be implemented using hardware, software, or combinations of hardwareand software. Also, where applicable, the various hardware componentsand/or software components set forth herein may be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein may be separated into sub-components comprising software,hardware, or both without departing from the scope of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components may be implemented as hardware components andvice-versa.

Software, in accordance with the present disclosure, such as programcode and/or data, may be stored on one or more computer readable media.It is also contemplated that software identified herein may beimplemented using one or more computers and/or computer systems,networked and/or otherwise. Where applicable, the ordering of varioussteps described herein may be changed, combined into composite steps,and/or separated into sub-steps to provide features described herein.

The foregoing disclosure is not intended to limit the present disclosureto the precise forms or particular fields of use disclosed. As such, itis contemplated that various alternate embodiments and/or modificationsto the present disclosure, whether explicitly described or impliedherein, are possible in light of the disclosure. For example, the aboveembodiments have focused on the user and user device, however, acustomer, a merchant, a service or payment provider may otherwisepresented with tailored information. Thus, “user” as used herein canalso include charities, individuals, and any other entity or personreceiving information. Having thus described embodiments of the presentdisclosure, persons of ordinary skill in the art will recognize thatchanges may be made in form and detail without departing from the scopeof the present disclosure. Thus, the present disclosure is limited onlyby the claims.

What is claimed is:
 1. A system comprising: a non-transitory memorystoring instructions; and one or more hardware processors coupled to thenon-transitory memory and configured to read the instructions from thenon-transitory memory to cause the system to perform operationscomprising: receiving, over a network connection, user input text from auser device; analyzing using a first learning model, the user inputtext; extracting, based on the analyzing, an emoji sequence related tothe user input text; processing, using a second learning model, theemoji sequence to obtain coordinate information for each emoji in theemoji sequence, wherein the processing comprises: determining acoordinate center for the each emoji in the emoji sequence, anddetermining a scale for the each emoji in the emoji sequence, whereinthe coordinate information comprises at least the coordinate center andthe scale for the each emoji in the emoji sequence; and combining atleast two of the emojis in the emoji sequence using the coordinateinformation and an emoji positioning model generated based on userfeedback for combining the emoji sequence, wherein the combiningcomprises: determining, using the emoji positioning model, a relativeposition of each of the at least two of the emojis to other one(s) ofthe at least two of the emojis based on the coordinate information andthe user feedback associated with the at least two of the emojis; andplacing the each of the at least two of the emojis together in an emojimashup based on the relative position(s).
 2. The system of claim 1,wherein the operations further comprise: transmitting, over the networkconnection, the emoji mashup to the user device, wherein the emojimashup relates to the user input text.
 3. The system of claim 1, whereinthe analyzing includes converting the user input text from words tovectors.
 4. The system of claim 3, wherein the operations furthercomprise filtering the converted vectors into matrices and generatingnew emojis based on the filtering.
 5. The system of claim 1, wherein theoperations further comprise training a text to emoji sequence model andconverting the user input text to emojis using the text to emojisequence model.
 6. The system of claim 1, wherein the coordinateinformation is obtained through object recognition from the secondlearning model.
 7. The system of claim 1, wherein the operations furthercomprise: transmitting, over the network connection, one or more of theemojis in the emoji sequence to the user device.
 8. A method comprising:receiving, over a network connection, text data from a user device;analyzing the text data; extracting, based on the analyzing, an emojisequence related to the text data; processing the emoji sequence toobtain coordinate information for each emoji in the emoji sequence,wherein the processing comprises: determining a coordinate center forthe each emoji in the emoji sequence, and determining a scale for theeach emoji in the emoji sequence, wherein the coordinate informationcomprises at least the coordinate center and the scale for the eachemoji in the emoji sequence; and combining at least two of the emojis inthe emoji sequence using the coordinate information and an emojipositioning model generated based on user feedback for combining theemoji sequence, wherein the combining comprises: determining, using theemoji positioning model, a position of each of the at least two of theemojis relative to other one(s) of the at least two of the emojis basedon the coordinate information and the user feedback associated with theat least two of the emojis; and placing the each of the at least two ofthe emojis together in an emoji mashup based on the position of the eachof the at least two of the emojis.
 9. The method of claim 8, furthercomprising: transmitting, over the network connection, the emoji mashupto the user device, wherein the emoji mashup relates to the text data.10. The method of claim 8, wherein the analyzing includes converting thetext data from words to vectors.
 11. The method of claim 10, wherein theanalyzing includes a making a sentiment analysis to determine a purposeand a context of the text data.
 12. The method of claim 8, furthercomprising training a text to emoji sequence model and converting thetext data to emojis using the text to emoji sequence model.
 13. Themethod of claim 8, wherein coordinate information is obtained throughobject recognition from a learning model.
 14. The method of claim 8,further comprising: transmitting, over the network connection, one ormore of the emojis in the emoji sequence to the user device for userselection.
 15. A non-transitory machine-readable medium having storedthereon machine readable instructions executable to cause a machine toperform operations comprising: receiving, over a network connection,user input text from a user device; analyzing the user input text;extracting, based on the analyzing, an emoji sequence related to theuser input text; processing the emoji sequence to obtain coordinateinformation for each emoji in the emoji sequence, wherein the processingcomprises: determining a coordinate center for the each emoji in theemoji sequence, and determining a scale for the each emoji in the emojisequence, wherein the coordinate information comprises at least thecoordinate center and the scale for the each emoji in the emojisequence; and combining at least two of the emojis in the emoji sequenceusing the coordinate information and an emoji positioning modelgenerated based on user feedback for combining the emoji sequence,wherein the combining comprises: determining, using the emojipositioning model, a relative position of each of the at least two ofthe emojis to other one(s) of the at least two of the emojis based onthe coordinate information and the user feedback associated with the atleast two of the emojis; and merging the each of the at least two of theemojis in an emoji mashup based on the relative position(s).
 16. Thenon-transitory machine-readable medium of claim 15, wherein theoperations further comprise: transmitting, over the network connection,the emoji mashup to the user device, wherein the emoji mashup relates tothe user input text.
 17. The non-transitory machine-readable medium ofclaim 15, wherein the analyzing includes converting the user input textfrom words to vectors.
 18. The non-transitory machine-readable medium ofclaim 17, wherein the operations further comprise filtering theconverted vectors into matrices and generating new emojis based on thefiltering.
 19. The non-transitory machine-readable medium of claim 15,wherein the operations further comprise training a text to emojisequence model and converting the user input text to emojis using thetext to emoji sequence model.
 20. The non-transitory machine-readablemedium of claim 15, wherein the coordinate information is obtainedthrough object recognition from a learning model.